Comparing and Improving the Accuracy of Nonprobability Samples: Profiling Australian Surveys

There has been a great deal of debate in the survey research community about the accuracy of nonprobability sample surveys. This work aims to provide empirical evidence about the accuracy of nonprobability samples and to investigate the performance of a range of post-survey adjustment approaches (calibration or matching methods) to reduce bias, and lead to enhanced inference. We use data from five nonprobability online panel surveys and com­pare their accuracy (pre- and post-survey adjustment) to four probability surveys, including data from a probability online panel. This article adds value to the existing research by assessing methods for causal inference not previously applied for this purpose and dem­onstrates the value of various types of covariates in mitigation of bias in nonprobability online panels. Investigating different post-survey adjustment scenarios based on the avail­ability of auxiliary data, we demonstrated how carefully designed post-survey adjustment can reduce some bias in survey research using nonprobability samples. The results show that the quality of post-survey adjustments is, first and foremost, dependent on the avail­ability of relevant high-quality covariates which come from a representative large-scale probability-based survey data and match those in nonprobability data. Second, we found little difference in the efficiency of different post-survey adjustment methods, and inconsis­tent evidence on the suitability of 'webographics' and other internet-associated covariates for mitigating bias in nonprobability samples

Do We Have to Mix Modes in Probability-Based Online Panel Research to Obtain More Accurate Results?

Online probability-based panels often apply two or more data collection modes to cover both the online and offline populations with the aim of obtaining results that are more representative of the population of interest. This study used such a panel to investigate how necessary it is, from the coverage error standpoint, to include the offline population by mixing modes in online panel survey research. This study evaluated the problem from three different perspectives: undercoverage bias, bias related to survey item topics and vari­able characteristics, and accuracy of online-only samples relative to nationally representa­tive benchmarks. The results indicated that attitudinal, behavioral, and factual differences between the online and offline populations in Australia are, on average, minor. This means that, considering that survey research commonly includes a relatively low proportion of the offline population, survey estimates would not be significantly affected if probability-based panels did not mix modes and instead were online only, for the majority of topics. The benchmarking analysis showed that mixing the online mode with the offline mode did not improve the average accuracy of estimates relative to nationally representative bench­marks. Based on these findings, it is argued that other online panels should study this issue from different perspectives using the approaches proposed in this paper. There might also be an argument for (temporarily) excluding the offline population in probability-based on­line panel research in particular country contexts as this might have practical implications

Approaches to dealing with survey errors in online panel research

Survey research is a relatively young field, and online surveys including online panel surveys are now routinely used for collecting survey data. We distinguish between different types of online panels, and this thesis is focused on both probability-based and nonprobability-based general population panels. To increase the quality of online panels in the era of nonresponse, more methodological research is needed, and that is the focus of the research in this thesis. To investigate approaches to dealing with survey errors, the Total Survey Error paradigm as a conceptual framework is applied, and both errors of representation and errors of measurement are the subject of this research. One of the contributions of this thesis is a review and discussion of a range of data sources and methodology which can be used in the study of survey errors. The other theoretical and practical contributions, presented within three groups, are related to the investigation of individual types of survey errors in online panel research. First, worldwide probability-based online panels are identified, and their methodological approaches to recruitment and data collection reviewed and compared as part of a meta-analysis. The study shows high levels of heterogeneity in both recruitment rates and recruitment solutions, as well as explains variability of recruitment rates. The other studies on errors of representation present evidence on how online panel paradata can be effectively transformed and used to identify about three in four nonrespondents in a subsequent panel wave, and answer the question of why people participate in online panel surveys while presenting evidence on how social-psychological theories can explain survey participation in a longitudinal design. Second, two studies focus on measurement error in probability-based online panel research due to mixing modes. The study on measurement mode effects shows how measurement error is present in the case of a lack of measurement equivalence between modes, and presents evidence on how applying matching methods (like coarsened exact matching) quite effectively controls for self-selection bias due to non-random assignment of online panellists to modes. The study on individual-level measurement mode effects presents a newly identified source of measurement error in online panel survey, that is, panel measurement mode effects. It also conceptualizes and showcases how panel conditioning can be a factor of two measurement aspects. These results are later related to a trade-off between representation (undercoverage) and measurement bias. Third, the thesis studies two cost- and time-efficient approaches to online data collection - nonprobability online panels and a fairly new combination of random digit dialing, text message invitations, and web-push methodology. The study on nonprobability panels, which are generally considered as less accurate but cheaper than probability-based panels, investigates post-survey adjustment methodology to improve inference in nonprobability samples. It presents evidence on how accuracy can be improved under different external data access scenarios. The study on a new approach to online survey data collection shows very low response rates, and outlines effective solutions to increase response (such as advance SMS and reminders). It also presents evidence on the fairly high accuracy of the proposed approach, which seems to be feasible for continuing recruitment to a probability-based online panel. In the final section of the thesis, the cost dimension of online survey research is discussed, the requirement of collecting data from the offline population in probability-based online panel research from different perspectives is challenged, and the theoretical contributions of this research are explained in more detail

A universal global measure of univariate and bivariate data utility for anonymised microdata

A universal global measure of univariate and bivariate data utility for anonymised microdata. This paper presents a new global data utility measure, based on a benchmarking approach. Data utility measures assess the utility of anonymised microdata by measuring changes in distributions and their impact on bias, variance and other statistics derived from the data. Most existing data utility measures have significant shortcomings – that is, they are limited to continuous variables, to univariate utility assessment, or to local information loss measurements. Several solutions are presented in the proposed global data utility model. It combines univariate and bivariate data utility measures, which calculate information loss using various statistical tests and association measures, such as two-sample Kolmogorov–Smirnov test, chi-squared test (Cramer’s V), ANOVA F test (eta squared), Kruskal-Wallis H test (epsilon squared), Spearman coefficient (rho) and Pearson correlation coefficient (r). The model is universal, since it also includes new local utility measures for global recoding and variable removal data reduction approaches, and it can be used for data protected with all common masking methods and techniques, from data reduction and data perturbation to generation of synthetic data and sampling. At the bivariate level, the model includes all required data analysis steps: assumptions for statistical tests, statistical significance of the association, direction of the association and strength of the association (size effect). Since the model should be executed automatically with statistical software code or a package, our aim was to allow all steps to be done with no additional user input. For this reason, we propose approaches to automatically establish the direction of the association between two variables using test-reported standardised residuals and sums of squares between groups. Although the model is a global data utility model, individual local univariate and bivariate utility can still be assessed for different types of variables, as well as for both normal and non-normal distributions. The next important step in global data utility assessment would be to develop either program code or an R statistical software package for measuring data utility, and to establish the relationship between univariate, bivariate and multivariate data utility of anonymised data

Panel mixed-mode effects: does switching modes in probability-based online panels influence measurement error?

Online probability-based panels often apply two or more data collection modes to cover online and offline populations, and to collect data from onliners who do not respond online in time to contribute to a given wave. As a result, offline/ online status can change during the life of the panel for some individuals, which can improve response rates and representativeness, but may cause increased measurement error. In this study, we use Life in Australia™ survey data and online panel paradata to identify respondents who switched modes; almost 4% of the whole panel was interviewed using both online and offline modes in the first 2-years, and almost one-third of those 4% switched mode more than once. We selected all repeated substantive survey items, identified any relevant changes in responses that could be explained with mode effects, and determined the effect of mode switching on changes to answers, controlling for panel conditioning, panel fatigue and sociodemographic characteristics of panellists. This study identified a limited number of panel mode effects from panellists switching modes of data collection over time. We found evidence of recency and some social desirability, and established that measurement error may be more common when the proportion of mode switchers is higher. Moreover, panel conditioning had an effect on the frequency of changing answers; respondents provided more stable answers if they were more conditioned. We conclude that combining mode effects with panel conditioning, as well as an increasing representation bias over time, may lead to less accurate estimations in longitudinal surveys

High resolution profile of inorganic aqueous geochemistry and key redox zones in an arsenic bearing aquifer in Cambodia

Arsenic contamination of groundwaters in South and Southeast Asia is a major threat to public health. In order to better understand the geochemical controls on the mobility of arsenic in a heavily arsenic-affected aquifer in northern Kandal Province, Cambodia, key changes in inorganic aqueous geochemistry have been monitored at high vertical and lateral resolution along dominant groundwater flow paths along two distinct transects. The two transects are characterized by differing geochemical, hydrological and lithological conditions. Arsenic concentrations in groundwater are highly heterogenous, and are broadly positively associated with iron and negatively associated with sulfate and dissolved oxygen. The observed correlations are generally consistent with arsenic mobilization by reductive-dissolution of iron (hydr)oxides. Key redox zones, as identified using groupings of the PHREEQC model equilibrium electron activity of major redox couples (notably ammonium/nitrite; ammonium/nitrate; nitrite/nitrate; dissolved oxygen/water) have been identified and vary with depth, site and season. Mineral saturation is also characterized. Seasonal changes in groundwater chemistry were observed in areas which were (i) sandy and of high permeability; (ii) in close proximity to rivers; and/or (iii) in close proximity to ponds. Such changes are attributed to monsoonal-driven surface-groundwater interactions and are consistent with the separate provenance of recharge sources as identified using stable isotope mixing models

The Impact of Academic Discipline on University Teaching and Pedagogical Training Courses

Professional development of university teachers in teaching, not only in research, has become an important topic in the development of higher education (HE) policies in recent years. Specific research on pedagogical development of university teachers remains limited, not taking into account academic disciplines and their impact on teachers' perceptions of different components of their teaching and training courses intended for their professional development. In this paper, we are presenting the differences among Slovenian teachers in different disciplines regarding their perception of teaching and their attitude towards pedagogical training courses (PTCs) with the results of an online survey study. Our findings show that teachers of soft sciences are more involved in PTCs than university teachers of hard sciences, except those from Health sub-discipline. Teachers of soft sciences, in comparison to those of hard sciences, generally perceive the quality of their own teaching as slightly higher, and they also have a slightly, but statistically significantly better attitude towards PTCs and teaching in general. Teachers from Health sub-discipline are generally more similar to soft sciences teachers than to hard sciences teachers in terms of different pedagogical dimensions studied. We can conclude that there is quite a clear distinction between teachers of soft sciences and teachers of certain hard sciences in their attitude and perception of different elements of university teaching and training courses in general (external heterogeneity). However, there is little separation between individual disciplines (internal homogeneity), which should be taken into account at least when introducing and implementing PTCs for teachers in different disciplines at institutional or national level

Increasing food insecurity severity is associated with lower diet quality

Abstract Objective: Food insecurity may reduce diet quality, but the relationship between food insecurity severity and diet quality is under-researched. This study aimed to examine the relationship between diet quality and severity of household food insecurity. Design: A cross-sectional, online survey used the United States Department of Agriculture Household Food Security Six-item Short Form to classify respondents as food secure or marginally, moderately or severely food insecure. The Australian Recommended Food Score (ARFS; scored 0–73) determined diet quality (ARFS total and sub-scale scores). Survey-weighted linear regression (adjusted for age, sex, income, education, location and household composition) was conducted. Setting: Tasmania, Australia. Participants: Community-dwelling adults (aged 18 years and over). Results: The mean ARFS total for the sample (n 804, 53 % female, 29 % aged > 65 years) was 32·4 (s d = 9·8). As the severity of household food insecurity increased, ARFS total decreased. Marginally food-insecure respondents reported a mean ARFS score three points lower than food-secure adults (B = –2·7; 95 % CI (–5·11, –0·34); P = 0·03) and reduced by six points for moderately (B = –5·6; 95 % CI (–7·26, –3·90); P < 0·001) and twelve points for severely food-insecure respondents (B = –11·5; 95 % CI (–13·21, –9·78); P < 0·001). Marginally food-insecure respondents had significantly lower vegetable sub-scale scores, moderately food-insecure respondents had significantly lower sub-scale scores for all food groups except dairy and severely food-insecure respondents had significantly lower scores for all sub-scale scores. Conclusions: Poorer diet quality is evident in marginally, moderately and severely food-insecure adults. Interventions to reduce food insecurity and increase diet quality are required to prevent poorer nutrition-related health outcomes in food-insecure populations in Australia